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Abstract 

The primary objective of this research was to examine the relationship between student 
achievement in mathematics and pedagogical approach used by middle school mathematics 
teachers, in the United States, who participated in the Third International Mathematics and 
Science Study. In this research, student achievement was explored at the item, rather than test, 
level with the thought that differences might be found only at this micro level. It was 
hypothesized that middle school mathematics students whose teachers utilized a more student- 
centered, or constructivist, pedagogical approach would have a higher probability of obtaining 
the correct answer to mathematics items that measured conceptual, rather than procedural, 
understanding. This hypothesis was explicitly tested using differential item functioning analyses. 
Results supported the hypothesis, although not as strongly as had been expected. 
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Introduction 

For decades the educational community has debated the strengths and weaknesses of a 
teacher-directed approach to instruction as opposed to a student-centered approach to 
instruction. As early as 1900, Dewey advocated the need for schools to consider the experiences 
students brought with them to the classroom and be “active centers of scientific insight into 
natural materials and processes” (Dewey, 1 900, p. 1 9). He criticized traditional education for 
emphasizing the rote memorization of rules and symbols while treating students as passive 
absorbers of information, rather than active vital beings whose knowledge is constantly being 
constructed and transformed. Soon after, Thorndike (1906) equated teaching with the ability to 
enable students to recall facts and information from memory and believed this was best 
accomplished by repetitive practice. He stated, “the active recall of a fact from within is, as a 
rule, better than its impression from without”(Thomdike, 1906, p. 123). While admitting that the 
teacher directed learning was being criticized for not promoting student understanding and not 
teaching students how to think, he argued that in some cases the best way to ensure a student was 
thinking was to give the student facts to think about. 

More recently, with the publication of the Curriculum and Evaluation Standards (NCTM, 
1989) and the Professional Standards for Teaching Mathematics (NCTM, 1991) [Standards], 
there is an implicit belief among most educators that a student-centered approach to teaching 
mathematics is superior to the more traditional “talk and chalk” or teacher-directed approach to 
teaching. These documents reflect what most mathematics educators feel empirical research 
supports in terms of what it means to know and understand mathematics, as well as how this 
knowledge of mathematics can best be acquired within a learning environment. According to the 
Standards “ what a student learns depends a great deal on how he or she has learned it” (NCTM, 
1989, p .5). If this is true, then students coming from different learning environments should 
theoretically possess different types of knowledge and understanding. For example, a student 
who learns mathematics in a teacher-directed classroom, from a teacher who believes that 
teachers must direct students’ thinking should possess a different type of knowledge and 
understanding than a student who learns mathematics in a student-centered classroom, from a 
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teacher who believes students should construct their own knowledge base by exploring, 
conjecturing, and reasoning. 

In the case of teacher-directed learning, students are thought of as a tabula rasa where 
learning is characterized as passive and receptive (Dengate & Lerman, 1995). When students are 
taught in this fashion, it is logical to assume that their knowledge is somewhat limited to what 
was shown to them and what they remembered. These students should have acquired only a 
procedural understanding of the mathematics being taught, knowing what to do to solve 
particular mathematics problems but not why they were doing it. Although some students taught 
in this manner would probably be capable of generalizing what they learn, enabling them to 
reason mathematically and creatively problem solve, the majority of students probably would not. 

In the case of student-centered learning, students’ knowledge would logically seem less 
limited, for even if a student did not recall how to solve a particular type of problem, these 
students would have learned they are capable of figuring it out on their own. These students 
should have acquired a conceptual understanding of the mathematics being taught, knowing not 
only what to do but why they were doing it. The conceptual understanding acquired by these 
students should enable them to apply their knowledge in new mathematical situations. In other 
words, the underlying distribution of conceptual understanding, including mathematical 
reasoning and problem solving abilities, for these students should be higher than the underlying 
distribution of students who are taught in a more teacher directed manner. 

One of the mainstream explanations for the occurrence of differential item functioning is 
that items are actually measuring more than one ability while the groups considered differ in their 
underlying distributions on these abilities, yet only one ability is reported (Ackerman, 1992; 
Ackerman & Evans, 1994). If some of the mathematics problems considered actually measure 
conceptual understanding, in addition to procedural understanding, then given the theoretical 
considerations, differential item functioning should occur for students who are grouped based on 
the pedagogical approach used by their teacher. Specifically, those students taught from a more 
student-centered environment should have a higher probability of obtaining the correct answer to 
items that are measuring conceptual understanding, in addition to procedural understanding. 

Much time, energy, and money is invested in pre-service and in-service teacher training, 
as well as in curriculum changes, which promotes the belief that a student-centered learning 
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environment is superior to a teacher-directed learning environment. However, presently and 
historically little empirical evidence can be found which supports this belief. This intent of this 
research is to explore the relationship between student achievement in mathematics and 
pedagogical approach used by middle school mathematics teachers. This will help to provide 
some much needed empirical evidence as to what learning environment is most conducive to 
student learning and understanding in mathematics 



Methodology 

Participants 

The students in this study consist of Population 2 students, from the USA, who 
participated in the Third International Mathematics and Science Study [TIMMS]. “Population 
2”, a term used by those who developed TIMMS refers to 13 year old students. These students 
were in the 7th or 8th grade at the time of testing. More specifically. Population 2 is defined as 
the two adjacent grades that will maximize coverage of 13 year old students (Martin & Kelly, 
1996). 

Instrumentation 

Measure of Mathematical Ability 

All of the multiple choice mathematics items from the TIMMS test administered to 
Population 2 students were used as the measure of mathematics ability. This test consists of 
eight different booklets, each consisting of 40 items pertaining to either mathematics or science. 
Only booklets one through seven were used. All mathematics items were classified in terms of 
content, performance expectations, and context. The content areas covered by items in the test 
include fractions and number sense, algebra, data representation, analysis and probability, 
measurement, and proportionality. Performance expectations include knowing, using routine 
procedures, problem solving, mathematical reasoning, proportionality, and communication 
(Robitaille et. al., 1993). Table 1 displays the number of items in each booklet which were 
classified into each content area. 
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Table 1 



Number of Mathematics Items in Each Booklet by Content Area 



Content area 


1 


2 


3 


Booklet 
4 5 


6 


7 


8 


Fractions and number sense 


11 


10 


11 


10 


10 


11 


11 


14 


Geometry 


5 


6 


6 


3 


6 


4 


5 


6 


Algebra 


8 


5 


6 


8 


4 


6 


6 


9 


Data representation, analysis and probability 


5 


4 


4 


6 


7 


6 


7 


5 


Measurement 


5 


5 


6 


4 


6 


4 


4 


3 


Proportionality 


3 


3 


4 


3 


6 


2 


4 


4 



Test Design 

A variant of matrix sampling was used in the TIMMS test design. In other words, not all 
items were administered to all examinees. Specifically, a subset of items was assigned to 
individual students so as to produce reliable estimates of the populations’ performance on all 
items, even thought no student has responded to all of the items (Martin & Kelley, 1996). This 
approach allows students to respond to a much smaller number of items than the total number of 
items in the pool, which, including both mathematics and science items, is equivalent to 530 
items. This approach eliminates the negative impacts associated with requiring students to spend 
the number of hours necessary to complete all the items, such as fatigue, decreased motivation, 
and reduced participation rates (Johnson, 1992). It has been estimated that it would take almost 
seven hours of testing for one student to complete all the items (Martin & Kelley, 1996). 
Moreover, the approach used also results in more efficient estimations of performance in 
subpopulations, in terms of errors of the estimates (Johnson, 1992). 

The TIMMS item pool was divided into 26 mutually exclusive clusters, each labeled by a 
different letter from the alphabet. Items in clusters A through H consist of mathematics and 
science multiple-choice type items. Items in clusters I through R consist of mathematics and 
science multiple-choice type items, as well as short-answer items. Items in clusters S through U 
consist of only mathematics extended-response type items. Items in cluster V consist of both 
mathematics and science multiple-choice type items, as well as short- answer and extended- 
response items. Clusters W through Z consist of only science extended-response type items 
(Marin & Kelley, 1996). These clusters were then arranged into eight booklets, each of which 
contains up to seven item clusters. Cluster A, the core cluster, appears in the second position in 
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every booklet, whereas the remaining clusters each appear in as few as one booklet, or as many 
as three. Booklet 8 consists only of items not present in other booklets and therefore was not 
analyzed in this study. 

The international median KR-20 coefficients across the eight test booklets for seventh 
grade students ranged from 0.91 in Hong Kong to 0.75 in Iran. The international median KR-20 
coefficients across the eight test booklets for eighth grade students ranged from 0.91 in Bulgaria 
to 0.73 in Kuwait. The median KR-20 for the United States was 0.89. The international median, 
which is the median of reliability coefficients for all countries, is 0.86. Table 2 provides the 
internal reliability for the subset of participating examinees used in this study for each of the 
seven booklets used. 



Table 2 



KR-20 Coefficients for Booklets 1 Through 7 



Booklet 


Sample size (n) 


KR-20 


1 


984 


0.86 


2 


986 


0.86 


3 


990 


0.85 


4 


987 


0.87 


5 


993 


0.87 


6 


960 


0.86 


7 


980 


0.88 



Teacher Questionnaire 

A survey, consisting of four sections, was administered to all participating teachers. The 
first section gathered demographic information of teachers, including the level of mathematics 
they teach, how often they teach, how much time they spend on activities related to teaching, 
beliefs about mathematics, familiarity with Standards documents, and how much time they are 
able to interact with other teachers. The second section consists of questions specific to the class 
being tested including, textbook used, limitations to teaching, calculator use, planning, as well as 
classroom activities. The third section consists of a set of exercises pertaining to various 
mathematical topics and questions teachers whether or not they have taught or will teach the 
content area covered by the given exercises to their class during the school year. The fourth 
section is designed to measure teachers’ beliefs about how mathematics is best taught. 




8 



Effect of Different Pedagogical Approaches Page 8 



Sixteen questions from the teacher survey were used to create a measure designed to 
differentiate teachers with opposing pedagogical beliefs. Appendix A contains the measure in its 
entirety. The items used were scored such that a low number represented a teacher directed 



pedagogical belief and a high number represented a student directed pedagogical belief. The 
lowest possible score a teacher could obtain was 16, while the highest possible score a teacher 
could obtain was 60. Actual scores ranged from 22 to 53. The reliability of this measure was 
0.80. Figure 1 illustrates the distribution of scores on this measure. Fifteen of the questions used 
were Likert-type survey questions while one of the questions used was a ranking task. Of the 
527 teachers administered the survey, 161 were eliminated because they either did not complete 
the survey at all, or only completed part of the survey. Many of these teachers may not have 
completed the items used because they taught science, rather than mathematics because only 374 
mathematics classes were tested. This would imply that only 8 mathematics teachers were 
eliminated from the study. 



30 



25 



20 




10 




o 



5 




fNjco^riotor-oocnoi-CNco 

CNCNCNCNCNCNCNCNCOCOCOCO 



tf) tO I s - 

CO CO CO CO 



Score 



Figure 1. Distribution of scores on the teacher survey. 
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Questions 1 through 1 5 used were Likert-type survey questions with either three or four response 
options. Questions 1 through 4 pertain to how often a teacher asks students to do some particular 
task. A response or ‘never or almost never’ was scored as 1, ‘some lessons’ was scored as 2, 
‘most lessons’ was scored as 3, and ‘every lesson was scored as 4. Questions 5 through 7 pertain 
to how important a teacher believes it is for students to possess certain mathematical skills in 
order for students to succeed in school mathematics. A response of ‘not important was scored as 
1, ‘somewhat important’ was scored as 2, and ‘very important’ was scored as 3. Questions 8 
through 15 pertain to the type of homework teachers assign. Those teachers who stated they did 
not assign homework were eliminated from the sample. A response of ‘never’ was scored as 1, 
‘rarely’ was scored as 2, ‘sometimes’ was scored as 3, and ‘always’ was scored as 4. It was not 
necessary to reverse-score any of the Likert-type items used. Missing responses for teachers who 
completed the entire survey were given the modal response for these Likert-type items. 

Question 16 was a ranking task. This question presented teachers with a classroom 
situation and asked teachers to rank order them, placing a ‘ 1 ’ in the box next to the approach they 
believed was best. This task was scored 1 if a teacher chose approach ‘a’, ‘b’, or ‘c’, as the best , 
approach, 2 if a teacher chose approach ‘d’ or ‘e’ as the best approach, and 3 if a teacher chose 
approach ‘f as the best approach. Teachers who ranked ‘a’, ‘b’, or ‘c’, as the best way to 
approach the topic can be considered to use more of a traditional pedagogical approach, because 
all of these approaches have the teacher telling or showing the students something. Although 
teachers who ranked ‘d’ or ‘e’ as the best way to approach the topic have the students working 
with the teachers, it is unclear how they will be working together, in terms of if the class is more 
student centered or teacher centered. However, it would seem that these approaches are more 
teacher-directed because the teacher is imposing their solution strategies onto the students. Those 
teachers who chose approach ‘f as the most ideal approach can be considered to be utilizing a 
more constructivist approach as they are allowing the students to try and solve the problem on 
their own. 

The 1 6-item chosen from the TIMMS teacher questionnaire to measure teachers’ 
pedagogical beliefs was used to group students. Student test data was linked to teacher 
questionnaire data using classroom identification numbers. Therefore, students whose teachers 
were eliminated from the sample were also eliminated from the sample. A stringent criterion 
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was set to label teachers as “constructivist”. Only teachers who scored above the 90th percentile 
were classified as “constructivist” teachers. For all booklets, approximately 89% of teachers 
obtained a score below 41 on the teacher measure. Therefore students whose teacher obtained a 
score above 41 on the teacher measure were considered as being from the “constructivist” or 
minority group, and students whose teacher obtained a score about 4 1 were considered the 
“teacher-directed” or majority group. 

Analyses 

Due to the complexity of the test design, analyses were conducted on each booklet 
separately. This strategy permits each item used in the mathematics achievement test of TIMMS 
to be used. It also allows cross-validation of results across different groups of examinees for 
items that appear in more than one booklet. These analyses were conducted using SIBTEST 
(Shealy & Stout, 1993) in an exploratory fashion to detect items which students from & student- 
centered classroom have a higher probability of obtaining a correct answer. Due to the design of 
the TIMMS study, it was possible to achieve conflicting results in the DIF analyses, such that an 
item that exhibited DIF in one booklet did not demonstrate DIF from another booklet. However, 
since the statistic provided by SIBTEST comes from a normal distribution with known variance, 
it was possible to average the test statistics provided by each analysis over the same items. 
Specifically, since the statistic calculated by SIBTEST is normally distributed, the mean of these 
statistics is also normally distributed. Therefore, the standard deviation of the mean of the three 
test statistics, obtained from testing the same item from three different booklets, can be obtained 
using normal distribution theory. This in turn allows one to calculate a test statistic for the mean 
of these test statistics, as the following proof demonstrates. 

Let P come from a normal distribution with variance a , then 

Var (P ) = Var (Pi +^2 +- + Pn) 

n 

= T-(Var([J|) + Var(P 2 )+■■■ + Var([J n )) 
n 

The variance of p j is not directly provided by SIBTEST; however, since the P statisticand the 
associated z-score are given the standard deviation is easily obtained by dividing the P statistic 
by the z-score associated with it. Squaring the standard deviation provides the variance of each 
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test statistic. In all cases when an item appeared in more than one booklet this approach was 
used. 

Results 

Table 3 illustrates the results of the differential item functioning analyses for the focus items that 
appeared in more than one booklet. None of the other items that appeared in only one booklet 
showed a statistically significant amount of DIF. Items labeled A1 through A6, appeared in all 
seven booklets (n = 7), while all other items appeared in only three booklets (n = 3). 
Unfortunately, the items that appear in more than one booklet are restricted making it impossible 
to include them in this paper. 



Table 3 

Average Test Statistics for Items Appearing in More than One Booklet 



Item 


P 


a 2 


z-score 


A1 


-0.057 


0.022 


- 2.577* 


A2 


0.026 


0.022 


1.156 


A3 


-0.013 


0.026 


-0.490 


A4 


-0.018 


0.021 


-0.843 


A5 


-0.010 


0.025 


-0.383 


A6 


0.032 


0.022 


1.463 


B1 


0.025 


0.026 


0.939 


B2 


0.014 


0.032 


0.427 


B3 


-0.011 


0.029 


-0.365 


B4 


-0.009 


0.025 


-0.362 


B5 


0.004 


0.027 


0.158 


B6 


-0.024 


0.024 


-0.992 


Cl 


0.046 


0.029 


1.592 


C2 


0.030 


0.026 


1.131 


C3 


-0.094 


0.028 


-3.407* 


C4 


-0.003 


0.033 


-0.092 


C5 


-0.010 


0.030 


-0.341 


C6 


-0.021 


0.027 


-0.777 



* p < 0.05 (table continues) 
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Table 3 (continued) 



Average Test Statistics for Items Appearing in More than One Booklet 



Item 


P 


a 2 


z-score 


D1 


0.014 


0.041 


0.331 


D2 


0.029 


0.025 


1.157 


D3 


0.016 


0.025 


0.621 


D4 


0.000 


0.029 


0.000 


D5 


-0.013 


0.025 


-0.519 


D6 


0.005 


0.030 


0.180 


El 


0.002 


0.031 


0.054 


E2 


0.007 


0.030 


0.243 


E3 


0.072 


0.033 


2.207* 


E4 


0.003 


0.033 


0.080 


E5 


-0.007 


0.027 


-0.269 


E5 


0.028 


0.025 


1.116 


FI 


-0.018 


0.028 


-0.650 


F2 


-0.021 


0.033 


-0.652 


F3 


0.043 


0.027 


1.592 


F4 


0.009 


0.025 


0.353 


F5 


-0.008 


0.029 


- 0.277 


F6 


-0.043 


0.025 


-1.742* 


G1 


-0.006 


0.030 


-0.189 


G2 


0.007 


0.026 


0.286 


G3 


-0.041 


0.029 


-1.405 


G4 


0.009 


0.026 


0.352 


G5 


0.042 


0.028 


1.523 


G6 


-0.001 


0.027 


-0.024 


HI 


-0.018 


0.027 


-0.655 


H2 


0.029 


0.023 


1.250 


H3 


0.022 


0.021 


1.040 


H4 


0.005 


0.030 


0.168 


H5 


-0.033 


0.029 


-1.117 


H6 


-0.033 


0.025 


1.293 



* p < 0.05 



Discussion 



Several items did show a statistically significant amount of DIF in favor of the focal 
group and these items were considered to be measuring more of a conceptual understanding of 
mathematics. For example, one of the items that showed a statistically significant amount of DIF 
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asked examinees to determine how many more pieces needed to be shaded in a given figure so 
that a certain fraction of the figure was shaded. However different denominators were used in the 
symbolic and pictorial representations of the fraction, implicitly requiring examinees to find a 
common denominator for the two different representations in order to successfully complete the 
problem. This would imply that examinees had to know not only how to convert fractions so that 
they had the same denominator, but why one would want to do so. Furthermore, the reason why 
one needed to find a common denominator in this situation was not the standard or typical reason 
why, because no computation was needed to complete the problem. 

A similar argument could be made for all of the other items that showed a statistically 
significant amount of DIF in favor of students taught in a more student-centered environment. 
However, this same argument could probably hold for some of the other items that did not 
exhibit a statistically significant amount of DIF but were thought to be measuring more of a 
conceptual understanding of mathematics, including mathematical reasoning. Furthermore, one 
item was found to be functioning differentially in favor of students who were taught in a more 
teacher-centered environment. So what does all this mean? It would seem that different 
pedagogical approaches have some effect on the way students view items, but not as drastic as 
had been expected. There are several different ideas that come to mind as to why this might be 
the case. 

First of all, one of the biggest limitations of this study is the method used to group 
examinees. Although the measure was fairly reliable, it relies on teachers’ self report as to what 
they are doing in the classroom. This report may or may not be accurate. In a qualitative study 
conducted by TIMMS researchers, it was found that few, if any, mathematics teachers in the 
United States truly teach in a constructivist manner. It is possible that the grouping method 
utilized affected the results, although it was the best that could be done with the given data. 
Secondly, it is difficult to determine just how much of an effect one teacher has on their students. 
Ideally, it would be nice to think that one good teacher who teaches mathematics to encourage 
conceptual understanding can eradicate any ill effects of previous teachers who focused more on 
procedural understanding of mathematics. However this is probably affected by many other 
factors. 




Effect of Different Pedagogical Approaches Page 14 



In conclusion, the main purpose of this research was to explore the relationship between 
pedagogical approach used by mathematics teachers and student achievement and to determine if 
items which were measuring more of a conceptual understanding of mathematics functioned 
differentially for examinees who were taught from contrasting pedagogical approaches. 
Theoretically it makes sense that this should be the case. However these results do not 
conclusively show this to be true or false. While several items did exhibit DIF in favor of 
students who were grouped into the student-centered group, many items did not. Furthermore, 
due to the grouping methodology used it is uncertain how contrasting the pedagogical approaches 
used by teachers in the two groups actually were. Future research is warranted to further explore 
these issues. 
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APPENDIX A 



TEACHER SURVEY 
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In your mathematics lessons, how often do you usually ask students to do the following? 



1 . explain the reasoning behind an 
idea 

2. represent and analyze relationships 
using tables, charts, or graphs 

3. work on problems for which there is no 
immediately obvious method or solution 

4. write equations to represent 

relationships 



never or 


some 


most 


every 


almost never 


lessons 


lessons 


lesson 


never or 


some 


most 


every 


almost never 


lessons 


lessons 


lesson 


never or 


some 


most 


every 


almost never 


lessons 


lessons 


lesson 


never or 


some 


most 


every 


almost never 


lessons 


lessons 


lesson 



If you assign mathematics homework, how often do you assign each of the following kinds of 
tasks? 



5. reading in a textbook or 
supplementary material 

6. writing definitions or other short 
writing assignment 

7. small investigation(s) or gathering 
data 

8. working individually on long term 
projects or experiments 

9. working as a small group on long 
term projects or experiments 

10. finding one or more uses of the 
content covered 

1 1 . preparing oral reports either 
individually or as a small group 

12. keeping a journal 



never 


rarely 


sometimes 


often 


never 


rarely 


sometimes 


often 


never 


rarely 


sometimes 


often 


never 


rarely 


sometimes 


often 


never 


rarely 


sometimes 


often 


never 


rarely 


sometimes 


often 


never 


rarely 


sometimes 


often 


never 


rarely 


sometimes 


often 



To be good at mathematics at school, how important do you think it is for students to... 



13. be able to think creatively 


not 


somewhat 


very 




important 


important 


important 


14. understand how mathematics is used 


not 


somewhat 


very 


in the real world 


important 


important 


important 


15. be able to provide reasons to support 


not 


somewhat 


very 


their solutions 


important 


important 


important 
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16 . 

Each year many teachers must help their students learn to solve problems such as “Juan was able 
to run 1 .5 kilometers in 5 minutes. If he was able to keep up this average speed, how far would 
he run in 12.5 minutes? ” If you needed to help your class solve such problems, what approach or 
sequence of approaches do you believe would best help students learn? 

Place a ‘ 1’ in the box in the right-hand margin next to the approach you believe to be the 
best. If you believe other approaches would also be acceptable, place a number in the box 
next to each one indicating the order in which you would consider using it. You need not 
choose more than one approach. Write zero in the box for any approach you do not 
consider acceptable. 



Teaching Approach 

I would present a general graph such as this because an understanding of graphs with a 
a constant ratio of change in x to change in y is one important mathematical tool for 

solving problems like this one. 

I would present the method of using proportional equations to solve this problem, as in: 
L5 = _x_ -> 5x = (1.5)(12.5) -> x = 18.75/5 = 3.75 km 
b 5 12.5 

After presenting other examples of this problem, I would assign practice exercises to 

students. 

c I would use the method suggested by the textbook for dealing with problems of this type, 

carrying out the strategy suggested by the textbook. 

I would work with students to develop a reasonable graph for this specific problem, such 
d as the one to the right and then work with students on using the properties of graphs like 

this one to find a numerical solution to the problem. 

I would have students use a calculator to find pairs of numbers that related to how long a 
e person has run at a constant average speed to how far that person has traveled. I would 
then have the students use these pairs of numbers to study how to determine the distance 

a person running at constant average speed would travel in a given time. 

I would divide the class into several groups and have the students in each group work 
f together on the problem until each group found a method for solving the given problem 
and then found a method that would work for similar problems. 
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